Sport training method and system and head-mounted VR device

ABSTRACT

A sport training method and system and a head-mounted VR device are disclosed. The method is performed by a head-mounted VR device. The head-mounted VR device is wirelessly connected to a hand-motion tracker and a foot-motion tracker. The method comprises: displaying a virtual demonstrative action; acquiring 6DOF head-motion data, 6DOF hand-motion data and 6DOF foot-motion data of the user; fusing the 6DOF hand-motion data and the 6DOF foot-motion data with the 6DOF head-motion data; calculating motion data of different parts of a human body according to the data that have been fused; and comparing the motion data of the different parts of the human body obtained by the calculation with standard sport training data of the virtual demonstrative action, and if a degree of similarity is greater than a threshold, determining that an action of a human-body part meets a training standard, and if not, displaying a corresponding prompting message in a virtual reality scene to prompt a human-body part whose action is not standard.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is entered as a bypass continuation application under35 U.S.C. § 111(a) of International Application No. PCT/CN2022/085321,filed on Apr. 6, 2022, which claims priority to Chinese PatentApplication No. 202110383654.7, filed on Apr. 9, 2021. The embodiment ofthe priority applications are hereby incorporated herein in theirentirety by reference.

TECHNICAL FIELD

The present application relates to the field of VR, in particular to asport training method and system and a head-mounted VR device.

BACKGROUND

With the advancement of society, people pay more and more attention tohealth. In the prior art, fitness activities (such as yoga) are usuallycarried out in a yoga club or gym. The participants are required to paya certain fee, and a yoga instructor carries out on-site teaching andinstructing. When there are a large number of participants, the yogainstructor cannot take care of every participant and instruct themindividually. On the other hand, if the class size is small, the costsof the participants and the yoga club will be increased. Thus, thetraditional ways of fitness have many problems, such as a lowintelligence degree, a low efficiency, a high cost, and a poor fitnesseffect.

In the prior art, users may also choose to practice at home bydownloading some videos of training subjects and learning and practicingon the computer or TV. However, this video-instruction method is only aone-way instruction, and users cannot obtain the feedback on whethertheir own actions are correct or accurate, cannot determine whethertheir yoga actions are standard, cannot correct their own actions intime, and cannot evaluate the fitness effect, which results in a lowefficiency and a poor fitness effect.

SUMMARY

In view of the problems in the conventional fitness methods, such as lowefficiency and effect, and the lack of systematic fitness evaluation andfeedback mechanism, the present application is proposed to provide asport training method, device and system to overcome the above problemsor at least partially solve the above problems.

According to an aspect of the present application, there is provided asport training method. The sport training method is performed by ahead-mounted VR device. The head-mounted VR device is wirelesslyconnected to a hand-motion tracker and a foot-motion tracker. Thehand-motion tracker is used to track motion data of a hand of a user,and the foot-motion tracker is used to track motion data of a foot ofthe user. The method comprises:

-   -   displaying a virtual demonstrative action;    -   acquiring 6DOF head-motion data, 6DOF hand-motion data and 6DOF        foot-motion data of the user;    -   fusing the 6DOF hand-motion data and the 6DOF foot-motion data        with the 6DOF head-motion data;    -   calculating motion data of different parts of a human body        according to the data that have been fused; and    -   comparing the motion data of the different parts of the human        body obtained by the calculation with standard sport training        data of the virtual demonstrative action, and if a degree of        similarity is greater than a threshold, determining that an        action of a human-body part meets a training standard, and if        not, displaying a corresponding prompting message in a virtual        reality scene to prompt a human-body part whose action is not        standard.

According to another aspect of the present application, a head-mountedVR device is provided. The head-mounted VR device is wirelesslyconnected to a hand-motion tracker and a foot-motion tracker. Thehand-motion tracker is used to track motion data of a hand of a user,and the foot-motion tracker is used to track motion data of a foot ofthe user. The head-mounted VR device comprises:

-   -   a displaying unit configured for displaying a virtual        demonstrative action;    -   an acquiring unit configured for acquiring 6DOF head-motion        data, 6DOF hand-motion data and 6DOF foot-motion data of the        user;    -   a fusing unit configured for fusing the 6DOF hand-motion data        and the 6DOF foot-motion data with the 6DOF head-motion data;    -   a calculating unit configured for calculating motion data of        different parts of a human body according to the data that have        been fused; and    -   a comparing unit configured for comparing the motion data of the        different parts of the human body obtained by the calculation        with standard sport training data of the virtual demonstrative        action, and if a degree of similarity is greater than a        threshold, determining that an action of a human-body part meets        a training standard, and if not, displaying a corresponding        prompting message in a virtual reality scene to prompt a        human-body part whose action is not standard.

According to yet another aspect of the present application, there isprovided a sport training system, which comprises: a server, and aplurality of head-mounted VR devices connected to the server via anetwork;

-   -   each of the head-mounted VR devices is configured for sending        motion data of different parts of a human body obtained by        calculation to the server;    -   the server is configured for receiving the motion data of the        different parts of the human body sent by the plurality of        head-mounted VR devices; in a virtual reality scene, according        to the motion data, drawing a plurality of user avatars, and        according to the plurality of user avatars, rendering to        generate a multi-person interactive motion scene; and sending        the multi-person interactive motion scene to each of the        head-mounted VR devices in real time; and each of the        head-mounted VR devices is further configured for displaying the        multi-person interactive motion scene on a display screen, so as        to realize remote interaction with another sport training user        in the virtual reality scene.

According to still another aspect of the present application, there isprovided a computer-readable storage medium having one or more programsstored therein, wherein when executed by a processor, the one or moreprograms implement the above-described sport training method.

By using the above technical solutions, the present application canachieve the following advantageous effects.

By wearing the hand-motion tracker and the foot-motion trackerrespectively at the specified parts of a hand and a foot of the user,and wearing the head-mounted VR device on the head of the user, themotion information of the key limbs of each of the sport training userscan be tracked in real time, and the user is not required to wear alarge number of sensors, which is very convenient for the user to wear.The head-mounted VR device acquires the 6DOF head-motion data collectedin real time by itself, the 6DOF hand-motion data collected in real timeby the hand-motion tracker and the 6DOF foot-motion data collected inreal time by the foot-motion tracker, and fuses the 6DOF hand-motiondata and the 6DOF foot-motion data with the 6DOF head-motion data toacquire high-precision high-frequency motion data, thereby increasingthe tracking precision. The motion data of different parts of the humanbody are calculated according to the data that have been fused, themotion data of the different parts of the human body obtained by thecalculation are compared with the standard sport training data, and ifthe degree of similarity is greater than a threshold, it is determinedthat the action of the human-body part meets the training standard, andif not, the prompting message is marked on the human-body part whoseaction is not standard in the head-mounted VR device for actionstandardizing correction. In this way, when the users are exercising athome, they can still judge whether the actions are standard or not, andcorrect non-demonstrative actions in time, thereby improving the fitnessefficiency and effect, and solving the problems in the process of yogaand other fitness activities, such as the inability to determine whetherthe actions are standard or not, the inability to correct the actions intime, and the lack of systematic fitness evaluation and feedbackmechanism. Moreover, the remote interaction among sport training userscan be realized by using the server, which further improves the userexperience.

The above description is merely a summary of the technical solutions ofthe present application. In order to more clearly know the elements ofthe present application to enable the implementation according to thecontents of the description, and in order to make the above and otherpurposes, features and advantages of the present application moreapparent and understandable, the particular embodiments of the presentapplication are provided below.

BRIEF DESCRIPTION OF THE DRAWINGS

By reading the following detailed description of the preferableembodiments, various other advantages and benefits will become clear toa person skilled in the art. The drawings are merely intended to showthe preferable embodiments, and are not to be considered as limiting thepresent application. Furthermore, throughout the drawings, the samereference signs denote the same elements. In the drawings:

FIG. 1 is a schematic flow chart of a sport training method according toan embodiment of the present application;

FIG. 2 is a schematic diagram of multi-user VR remote interactionaccording to an embodiment of the present application;

FIG. 3 is a schematic structural diagram of a sport training deviceaccording to an embodiment of the present application;

FIG. 4 is a schematic diagram of a wearing effect according to anembodiment of the present application;

FIG. 5 is a schematic structural diagram of a sport training systemaccording to an embodiment of the present application; and

FIG. 6 is a schematic structural diagram of a head-mounted VR deviceaccording to an embodiment of the present application.

DETAILED DESCRIPTION

The exemplary embodiments of the present application will be describedin further detail below with reference to the drawings. Although thedrawings illustrate the exemplary embodiments of the presentapplication, it should be understood that the present application may beimplemented in various forms, which should not be limited by theembodiments illustrated herein. In contrast, the purpose of providingthose embodiments is to more clearly understand the present application,and to completely convey the scope of the present application to aperson skilled in the art.

The virtual reality (VR) technique provides people a sense ofenvironmental immersion by simulating a virtual environment by using acomputer. In the present application, the problems in the process ofyoga and other fitness activities, such as the inability to determinewhether the actions are standard, the inability to correct the actionsin time, and the lack of systematic fitness evaluation and feedbackmechanism, are solved by combining the VR device with sensors, therebybringing the user a better fitness experience. In the technicalsolutions of the present application, before performing the sporttraining the user wears a hand-motion tracker and a foot-motion trackerrespectively at specified parts of the hand and the foot, and wears ahead-mounted VR device on the head. The hand-motion tracker and thefoot-motion tracker are wirelessly connected to the head-mounted VRdevice. During the sport training, the user makes corresponding actionsaccording to the virtual demonstrative actions displayed in thehead-mounted VR device.

FIG. 1 is a schematic flow chart of the sport training method accordingto an embodiment of the present application. As shown in FIG. 1 , themethod is performed by a head-mounted VR device, and comprises:

Step S100: displaying a virtual demonstrative action.

Step S110: acquiring 6DOF head-motion data, 6DOF hand-motion data and6DOF foot-motion data of the user.

6-DOF refers to that the information of 6 degrees of freedom (DOF) isused to represent the motion of an object, including the displacementinformation on X, Y and Z axes and the rotation information on the X, Yand Z axes. In the present embodiment, not only the change of the visualfield angle caused by the rotation of the parts of the body can bedetected, but also the change of up, down, front, back, left and rightdisplacements caused by the movement of the parts of the body can bedetected.

In the present embodiment, the 6-DOF motion data of the head, the handand the foot of the user are collected in real time to comprehensivelymonitor the motion data of the joints of the human body, which can moretruly and accurately reflect the motion of the human body.

Step S120: fusing the 6DOF hand-motion data and the 6DOF foot-motiondata with the 6DOF head-motion data.

In the present embodiment, the 6-DOF motion data of the head, the handand the foot are acquired by using the head-mounted VR device, thehand-motion tracker and the foot-motion tracker respectively; in otherwords, each of the 6-DOF motion data has its own coordinate system.Therefore, it is necessary to fuse the motion data of the head, the handand the foot, and particularly, to unify the 6-DOF motion data of thehand and the foot into the coordinate system of the head, so as to makethe motion of the real human body consistent with the motion of thevirtual human body in the head-mounted VR device.

Step S130: calculating motion data of different parts of a human bodyaccording to the data that have been fused.

In the present embodiment, the motion data of the head, the hand and thefoot are fused, the 6-DOF motion data of the hand and the foot areunified into the coordinate system of the head, and the motion data ofthe 6-DOF motion data of the head, the hand and the foot in the headcoordinate system are calculated.

Step S140: comparing the motion data of the different parts of the humanbody obtained by the calculation with standard sport training data ofthe virtual demonstrative action, and if a degree of similarity isgreater than a threshold, determining that an action of a human-bodypart meets a training standard, and if not, displaying a correspondingprompting message in a virtual reality scene to prompt a human-body partwhose action is not standard.

In the present embodiment, the standard sport training data are astandard action database established by collecting the data of thecorresponding actions completed by senior sport trainers according tothe above steps. For example, the threshold may be set to 95%.

In sum, in the technical solution of the present embodiment, by wearingthe hand-motion tracker and the foot-motion tracker respectively at thespecified parts of a hand and a foot of the user, and wearing thehead-mounted VR device on the head of the user, the motion informationof the key limbs of each of the sport training users can be tracked inreal time, and the user is not required to wear a large number ofsensors, which is very convenient for the user to wear. The head-mountedVR device acquires the 6DOF head-motion data collected in real time byitself, the 6DOF hand-motion data collected in real time by thehand-motion tracker and the 6DOF foot-motion data collected in real timeby the foot-motion tracker, and fuses the 6DOF hand-motion data and the6DOF foot-motion data with the 6DOF head-motion data to acquirehigh-precision high-frequency motion data, thereby increasing thetracking precision. The motion data of different parts of the human bodyare calculated according to the data that have been fused, the motiondata of the different parts of the human body obtained by thecalculation are compared with the standard sport training data, and ifthe degree of similarity is greater than a threshold, it is determinedthat the action of the human-body part meets the training standard, andif not, the prompting message is marked on the human-body part whoseaction is not standard in the head-mounted VR device for actionstandardizing correction. In this way, when the users are exercising athome, they can still judge whether the actions are standard or not, andcorrect non-demonstrative actions in time, thereby improving the fitnessefficiency and effect, and solving the problems in the process of yogaand other fitness activities, such as the inability to determine whetherthe actions are standard or not, the inability to correct the actions intime, and the lack of systematic fitness evaluation and feedbackmechanism.

In an embodiment of the present application, the head-mounted VR deviceis provided therein with a head-mounted tracking camera, anelectromagnetic signal receiving module or an ultrasonic signalreceiving module. The hand-motion tracker and the foot-motion trackerare both provided therein with an electromagnetic sensor and anelectromagnetic signal emitting module, or both provided therein with anultrasonic sensor and an ultrasonic signal emitting module.

The head-mounted VR device acquires the 6DOF head-motion data collectedin real time, the 6DOF hand-motion data collected in real time by thehand-motion tracker, and the 6DOF foot-motion data collected in realtime by the foot-motion tracker, which further comprises:

the head-mounted VR device collects the pose information of the headmotion in real time by using the built-in head-mounted tracking camera;

the hand-motion tracker and the foot-motion tracker collect the poseinformation of the hand motion and the pose information of the footmotion in real time respectively by using the built-in electromagneticsensor, and wirelessly transmit them to the electromagnetic signalreceiving module of the head-mounted VR device by using theelectromagnetic signal emitting module; or, the hand-motion tracker andthe foot-motion tracker collect the pose information of the hand motionand the pose information of the foot motion in real time respectively byusing the built-in ultrasonic sensor, and wirelessly transmit them tothe ultrasonic signal receiving module of the head-mounted VR device byusing the ultrasonic signal emitting module.

In the present embodiment, preferably, the head-mounted VR device isprovided with built-in high-performance central processing unit (CPU)and graphics processing unit (GPU), which may be a high-performancemobile platform such as Qualcomm snapdragon 845 and Qualcomm snapdragon865. The head-mounted VR device is further provided therein with 4head-mounted tracking cameras, and each of the head-mounted trackingcameras is configured as follows: (1) frame rate: 30 Hz or higher; (2)angle of view (FOV): 130° *80° (H*V) or higher; (3) exposure mode:generally global shutter; and (4) light transmission band of lens:generally about 400-900. The head-mounted VR device collects the 6-DOFpose information of the user's head motion relative to the environmentin real time by using the head-mounted tracking cameras.

In an embodiment of the present application, the head-mounted VR deviceis further provided therein with a first wireless-communication moduleand a first IMU sensor, and the hand-motion tracker and the foot-motiontracker are both further provided therein with a secondwireless-communication module and a second IMU sensor.

The head-mounted VR device acquires the 6DOF head-motion data collectedin real time, the 6DOF hand-motion data collected in real time by thehand-motion tracker, and the 6DOF foot-motion data collected in realtime by the foot-motion tracker, which further comprises:

the head-mounted VR device further collects the IMU information of thehead motion in real time by using the built-in first IMU sensor, andfuses the IMU information of the head motion with the pose informationof the head motion to obtain the 6DOF head-motion data;

the hand-motion tracker and the foot-motion tracker further collect theIMU information of the hand motion and the IMU information of the footmotion in real time respectively by using the built-in second IMUsensor, and wirelessly transmit them to the first wireless-communicationmodule of the head-mounted VR device by using the secondwireless-communication module; and

the head-mounted VR device fuses the pose information of the hand motionwith the IMU information of the hand motion to obtain the 6DOFhand-motion data, and fuses the pose information of the foot motion withthe IMU information of the foot motion to obtain the 6DOF foot-motiondata.

In addition to the 4 built-in head-mounted tracking cameras, in order tooutput the 6-DOF motion data with a high frequency (>200 Hz), thehead-mounted VR device generally requires to be provided with a built-inhigh-precision high-frequency IMU (Inertial Measurement Unit) inertialnavigation sensor to measure the three-axis attitude angle (or angularvelocity) and the acceleration of the head. High-frequency 6-DOFinformation of the motion state of the head-mounted VR device can beoutput by using the technique of simultaneous localization and mapping(SLAM) in combination with the IMU information.

The hand-motion tracker can realize the hand motion tracking by using anelectromagnetic sensor or an ultrasonic sensor. If an electromagneticsensor is used, an electromagnetic signal emitting module is provided inthe hand-motion tracker, and an electromagnetic signal receiving moduleis provided in the head-mounted VR device. If an ultrasonic sensor isused, an ultrasonic signal emitting module is provided in thehand-motion tracker, and an ultrasonic signal receiving module isprovided in the head-mounted VR device. The hand-motion tracker isfurther provided therein with a wireless-communication module. In orderto improve the tracking stability of the hand-motion tracker, it isfurther provided therein with an IMU sensor with a high precision and ahigh frequency to transmit the IMU inertial-navigation information inthe hand-motion tracker to the head-mounted VR device by using thewireless-communication module. The electromagnetic signal receivingmodule or the ultrasonic signal receiving module at the head-mounted VRdevice receives the motion electromagnetic signal or the motionultrasonic signal of the hand-motion tracker, combined with the IMUinformation of the hand motion tracking, the 6-DOF information of thehand-motion tracker is calculated in real time by using anelectromagnetic-signal processing algorithm or an ultrasonic-signalprocessing algorithm. The built-in motion sensor of the foot-motiontracker is the same as that of the hand. If the hand-motion tracker isprovided therein with an electromagnetic-wave sensor, the foot-motiontracker is also provided therein with an electromagnetic-wave sensor. Ifthe hand-motion tracker is provided therein with an ultrasonic sensor,the foot-motion tracker is also provided therein with an ultrasonicsensor.

In the present embodiment, the hand-motion tracker and the foot-motiontracker respectively acquire the 6DOF hand-motion data and the 6DOFfoot-motion data by using the built-in electromagnetic sensor orultrasonic sensor in combination with the second IMU sensor, and thehead-mounted VR device acquires the 6DOF head-motion data by using thehead-mounted tracking camera and the first IMU sensor, therebyovercoming the defects in the prior art that the acquirement of the usermotion data by using the IMU sensor can only acquire the 3-DOF attitudedata of the human joint points and cannot acquire the displacement dataof the joint points, and that the IMU sensor will have a cumulativedrift error over time. Thus, it acquires high-precision high-frequency6-DOF motion data and increases the tracking precision, so that theposition offset error can reach the order of millimeters, the rotationangle offset error can be controlled within 1.5°, and the outputtedfrequency is greater than 200 Hz.

In an embodiment of the present application, the head-mounted VR devicefuses the 6DOF hand-motion data and the 6DOF foot-motion data with the6DOF head-motion data, which comprises:

the head-mounted VR device acquires a rotation matrix and a translationvector of the electromagnetic signal receiving module or ultrasonicsignal receiving module relative to the first IMU sensor, performscoordinate system conversion on the 6DOF hand-motion data and 6DOFfoot-motion data by using the rotation matrix and the translationvector, and converts them into 6-DOF motion data with the first IMUsensor as the origin.

In the present embodiment, the hand-motion tracker and the foot-motiontracker are provided therein with the electromagnetic sensor or theultrasonic sensor, and the VR head-mounted displaying all-in-one machineis provided therein with the electromagnetic signal receiving module orthe ultrasonic signal receiving module. The coordinate systems of the6-DOF motion data of the hand and the foot use the receiving-end sensorsas the origin, while the coordinate system of the 6-DOF motion data ofthe head uses the first IMU sensor as the origin. Only by unifying the6-DOF motion data of the hand, the foot and the head into the samecoordinate system can the motion of the real human body be consistentwith that of the virtual human body in the head-mounted VR device.

Particularly, the 6-DOF motion data of the hand and the foot are unifiedinto the head coordinate system, which comprises: acquiring respectivelya rotation matrix R and a translation vector T of the electromagneticsignal receiving module or the ultrasonic signal receiving module in thehead-mounted VR device relative to the first IMU sensor, performingcoordinate system conversion on the 6-DOF motion data of the hand andthe foot by using the rotation matrix R and the translation vector T,and converting them into 6-DOF tracking data with the first IMU sensorbuilt in the head-mounted VR device as the origin. The formulas of thecoordinate system conversion are:6-DOF motion data with the first IMU sensor as the origin=R*6DOFhand-motion data+T  (1)6-DOF motion data with the first IMU sensor as the origin=R*6DOFfoot-motion data+T  (2)

In order to accurately calculate the motion data of the different partsof the human body, in an embodiment of the present application, beforesport training, the human-body motion data are calibrated according tothe actions made by the user corresponding to the demonstrative actions.Particularly, when the user wears the hand-motion tracker, thefoot-motion tracker and the head-mounted VR device, and makescorresponding actions according to the virtual calibration actionsdisplayed in the head-mounted VR device, the motion speed and the motionangle of the different parts of the human body, the displacementinformation of each part, and the motion hold time of each key part ofthe human body are calculated according to the data that have been fusedand the sport training calibration data by using an inverse kinematicsalgorithm.

Taking yoga as an example, the operation of the system according to thepresent application is described below.

(1) Before the exercise, the user firstly, according to a yoga traininginstruction manual, wears the motion tracking sensors on the specifiedparts of the hand and the foot and then wears the head-mounted VRdevice.

(2) After the user has worn the sensors, the calibration of thehuman-body motion data is performed firstly in a “yoga traininginstruction” APP in the head-mounted VR device. The user, according tothe demonstrative actions indicated by the APP, makes severalcorresponding actions, to perform the calibration of the human-bodymotion data, so as to make the motion tracking data of each sensor forthe user's yoga actions more stable and precise. The human-body motioncalibration data will be stored in the head-mounted VR deviceaccordingly. The whole calibration process spends about 1-2 minutes. Inorder to make the motion tracking data of the yoga actions more stableand precise, it is recommended to calibrate the human-body motion dataevery time after wearing the motion tracking sensors.

(3) After the above steps (1) and (2) have been completed, yoga traininginstruction may be provided. The user may select different yoga exerciseactions in the APP. After selecting the yoga exercise actions, the usercan see the virtual demonstrative action of each of the actions in thehead-mounted VR device, and then the user makes the correspondingactions simultaneously. The system will calculate the motion speed andthe motion angle of the different parts of the human body, thedisplacement information of each part, and the motion hold time of eachkey part of the human body according to the 6DOF head-motion data, the6DOF hand-motion data, the 6DOF foot-motion data and the human-bodymotion calibration data at this moment in combination with the IK(inverse kinematics) algorithm, and compare them with the yoga trainingdatabase in the head-mounted VR device. If the degree of similarity isgreater than 95%, it is considered that the action of the human-bodypart meets the yoga training standard, and if not, the human-body partwhose action is not standard is marked with a prompting message in thehead-mounted VR device for action standardizing correction.

(4) According to step (3), the actions in exercise courses are made inturn. The duration of each exercise course is about 30-40 minutes. Aftereach exercise course has ended, the APP will count the quantity of wrongactions, the occurrence time of wrong actions, the average success rateof correct actions, and the quantity of actions completed in thiscourse.

In an embodiment of the present application, the head-mounted VR devicefurther sends the motion data of the different parts of the human bodyobtained by the calculation to a server, and receives the multi-personinteractive motion scene rendered and updated by the server by using agraphics processing unit (GPU), so as to realize remote interaction withanother sport training user in the virtual reality scene.

As shown in FIG. 2 , the present embodiment requires to construct aserver. The user may determine the processing capacity, the applicationscene rendering capacity and other hardware configuration specificationsof the server according to the quantity of the clients in the actualapplication (i.e. the head-mounted VR devices shown in FIG. 2 ) and therendering complexity of the VR content. In the present embodiment, themaximum quantity of the clients that the server can support is 100.

In the present embodiment, the head-mounted VR device is providedtherein with components such as a CPU, a GPU, and a wireless networkmodule, collects the 6DOF head-motion data in real time, acquires the6DOF hand-motion data collected in real time by the hand-motion trackerand the 6DOF foot-motion data collected in real time by the foot-motiontracker, fuses the 6DOF hand-motion data and the 6DOF foot-motion datawith the 6DOF head-motion data, and calculates the motion data ofdifferent parts of the human body according to the data that have beenfused, to obtain 150 Hz of the motion data of the different parts of thehuman body. Each of the clients is connected to a network via a wirelessnetwork processor such as a wireless router, and then is connected tothe server via the network.

The server receives the 150 Hz of the motion data of the different partsof the human body sent by each of the head-mounted VR devices via thenetwork in real time, fuses the received motion data again, draws useravatars in the virtual reality scene, drives the virtual user avatars inreal time by using the GPU for the user data of each of the head-mountedVR devices, renders and updates the information content of themulti-person interactive scene, and then transmits it to thehead-mounted VR device of each of the clients in real time. Other sporttraining users may observe the motion states of the other sport trainingusers in the multi-person interactive motion scene in real time on theirown display screens from a third-party's perspective.

The present embodiment further constructs a main controller. The maincontroller serves as the administrator in the sport training system andis configured to manage the head-mounted VR devices in the system. Inthe present embodiment, the physical structure of the main controller isthe same as the physical structure of the head-mounted VR devices.

Taking yoga as an example, the operation of the user interaction systemaccording to the present application is described below.

(1) Before the exercise, the user sends a “request for joiningmulti-person fitness interaction” to the server in the multi-personfitness APP of the head-mounted VR device.

(2) The server quickly sends the instruction of “request for joiningmulti-person fitness interaction” sent by the user to the maincontroller.

(3) The main controller confirms the user's joining request. If the useris allowed to join, the main controller sends a consent instruction toenable the user to join the multi-person fitness system.

In the present embodiment, the main controller may be any senior fitnessexpert user who is responsible for inviting other fitness enthusiasticusers to join the multi-person fitness system, and is also responsiblefor taking normative measures such as establishing fitness rules andpunishment and reward mechanisms.

In the process of the sport training, any sport training user may chatwith the other sport training users by using a microphone on thehead-mounted VR device, and may share the technical methods of fitnesssports. Some senior fitness experts may help some junior sport trainingusers correct their technical actions of fitness, and may share someinteresting and active interactive topics conducive to fitness, such asdiet precautions of fitness sports, to realize the remote interactionamong the sport training users and improve the user experience.

FIG. 3 is a schematic structural diagram of a sport training deviceaccording to an embodiment of the present application. As shown in FIG.3 , the sport training device 300 comprises: a head-mounted VR device310, a hand-motion tracker 320 and a foot-motion tracker 330. Thehand-motion tracker 320 and the foot-motion tracker 330 are wirelesslyconnected to the head-mounted VR device 310. Before sport training, thehand-motion tracker 320 and the foot-motion tracker 330 are respectivelyworn on the specified parts of a hand and a foot of the user, and thehead-mounted VR device 310 is worn on the head of the user. The wearingeffect is shown in FIG. 4 . During the sport training, the user makescorresponding actions according to the virtual demonstrative actionsdisplayed in the head-mounted VR device 310.

The hand-motion tracker 320 is used to collect 6DOF hand-motion data inreal time and wirelessly transmit it to the head-mounted VR device 310.

The foot-motion tracker 330 is used to collect 6DOF foot-motion data inreal time and wirelessly transmit it to the head-mounted VR device 130.

The head-mounted VR device 310 is used to, during the sport training,display virtual demonstrative actions, collect 6DOF head-motion data inreal time, and acquire the 6DOF hand-motion data collected in real timeby the hand-motion tracker 320 and the 6DOF foot-motion data collectedin real time by the foot-motion tracker 330; fuse the 6DOF hand-motiondata and the 6DOF foot-motion data with the 6DOF head-motion data;calculate motion data of different parts of the human body according tothe data that have been fused; and compare the motion data of thedifferent parts of the human body obtained by the calculation with thestandard sport training data, and if the degree of similarity is greaterthan a threshold, determine that the action of a human-body part meets atraining standard, and if not, the head-mounted VR device 310 displays acorresponding prompting message in a virtual reality scene to prompt ahuman-body part whose action is not standard.

In an embodiment of the present application, the head-mounted VR device310 is provided therein with a head-mounted tracking camera, anelectromagnetic signal receiving module or an ultrasonic signalreceiving module; and the hand-motion tracker 320 and the foot-motiontracker 330 are both provided therein with an electromagnetic sensor andan electromagnetic signal emitting module, or are both provided thereinwith an ultrasonic sensor and an ultrasonic signal emitting module.

The head-mounted VR device 310 is particularly used to collect the poseinformation of the head motion in real time by using the built-inhead-mounted tracking camera.

The hand-motion tracker 320 and the foot-motion tracker 330 areparticularly used to collect the pose information of the hand motion andthe pose information of the foot motion in real time respectively byusing the built-in electromagnetic sensors, and wirelessly transmit themto the electromagnetic signal receiving module of the head-mounted VRdevice 310 by using the electromagnetic signal emitting modules.

Alternatively, the hand-motion tracker 320 and the foot-motion tracker330 collect the pose information of the hand motion and the poseinformation of the foot motion in real time respectively by using thebuilt-in ultrasonic sensors, and wirelessly transmit them to theultrasonic signal receiving module of the head-mounted VR device 310 byusing the ultrasonic signal emitting modules.

In an embodiment of the present application, the head-mounted VR device310 is further provided therein with a first wireless-communicationmodule and a first IMU sensor, and the hand-motion tracker 320 andfoot-motion tracker 330 are both further provided therein with a secondwireless-communication module and a second IMU sensor.

The head-mounted VR device 310 is further particularly used to collectthe IMU information of the head motion in real time by using thebuilt-in first IMU sensor, and fuse the IMU information of the headmotion with the pose information of the head motion to obtain the 6DOFhead-motion data.

The hand-motion tracker 320 and the foot-motion tracker 330 are furtherparticularly used to collect the IMU information of the hand motion andthe IMU information of the foot motion in real time respectively byusing the built-in second IMU sensors, and wirelessly transmit them tothe first wireless-communication module of the head-mounted VR device310 by using the second wireless-communication modules. The head-mountedVR device 310 is further used to fuse the pose information of the handmotion with the IMU information of the hand motion to obtain the 6DOFhand-motion data, and fuse the pose information of the foot motion withthe IMU information of the foot motion to obtain the 6DOF foot-motiondata.

In an embodiment of the present application, the head-mounted VR device310 is particularly used to acquire a rotation matrix and a translationvector of the electromagnetic signal receiving module or ultrasonicsignal receiving module relative to the first IMU sensor, performcoordinate system conversion on the 6DOF hand-motion data and the 6DOFfoot-motion data by using the rotation matrix and the translationvector, and convert them into 6-DOF motion data with the first IMUsensor as the origin.

In an embodiment of the present application, before sport training,after wearing the hand-motion tracker 320, foot-motion tracker 330 andhead-mounted VR device 310, the user makes corresponding actionsaccording to the virtual calibration action displayed in thehead-mounted VR device. The head-mounted VR device 310 is further usedto display the virtual calibration action before sport training andacquire sport training calibration data; and calculate the motion speedand the motion angle of the different parts of the human body, thedisplacement information of each part, and the motion hold time of eachkey part of the human body according to the data that have been fusedand the sport training calibration data by using an inverse kinematicsalgorithm.

FIG. 6 is a schematic structural diagram of a head-mounted VR deviceaccording to an embodiment of the present application. As shown in FIG.6 , the head-mounted VR device 310 comprises:

-   -   a displaying unit 3101 configured for displaying a virtual        demonstrative action;    -   an acquiring unit 3102 configured for acquiring 6DOF head-motion        data, 6DOF hand-motion data and 6DOF foot-motion data of the        user;    -   a fusing unit 3103 configured for fusing the 6DOF hand-motion        data and the 6DOF foot-motion data with the 6DOF head-motion        data;    -   a calculating unit 3104 configured for calculating motion data        of different parts of a human body according to the data that        have been fused; and    -   a comparing unit 3105 configured for comparing the motion data        of the different parts of the human body obtained by the        calculation with standard sport training data of the virtual        demonstrative action, and if a degree of similarity is greater        than a threshold, determining that an action of a human-body        part meets a training standard, and if not, displaying a        corresponding prompting message in a virtual reality scene to        prompt a human-body part whose action is not standard.

When acquiring the 6DOF head-motion data of the user, the acquiring unit3102 is particularly configured for:

-   -   acquiring pose information of the head-mounted VR device by        using a tracking camera of the head-mounted VR device;    -   acquiring IMU information of the head-mounted VR device by using        a first IMU sensor of the head-mounted VR device; and    -   fusing the pose information of the head-mounted VR device with        the IMU information of the head-mounted VR device to obtain the        6DOF head-motion data of the user.

The hand-motion tracker and the foot-motion tracker collect the poseinformation of the hand motion and the pose information of the footmotion in real time respectively by using a built-in electromagneticsensor/ultrasonic sensor, and wirelessly transmit to an electromagneticsignal receiving module/ultrasonic signal receiving module of thehead-mounted VR device by using an electromagnetic signal emittingmodule/ultrasonic signal emitting module.

The hand-motion tracker and the foot-motion tracker also collect IMUinformation of the hand motion and IMU information of the foot motion inreal time respectively by using a built-in second IMU sensor, andwirelessly transmit to a first wireless-communication module of thehead-mounted VR device by using a second wireless-communication module.

When acquiring the 6DOF hand-motion data/6DOF foot-motion data of theuser, the acquiring unit 3102 is particularly configured for:

-   -   receiving hand pose information and hand IMU information sent by        the hand-motion tracker, and fusing the hand pose information        and the hand IMU information to obtain the 6DOF hand-motion        data;    -   and/or,    -   receiving foot pose information and foot IMU information sent by        the foot-motion tracker, fusing the foot pose information and        the foot IMU information to obtain the 6DOF foot-motion data.

In an embodiment, the fusing unit 3103 is particularly configured for:acquiring a rotation matrix and a translation vector of a built-inelectromagnetic signal receiving module or ultrasonic signal receivingmodule of the head-mounted VR device relative to the first IMU sensor,and performing coordinate system conversion on the 6DOF hand-motion dataand the 6DOF foot-motion data by using the rotation matrix and thetranslation vector, to convert the 6DOF hand-motion data and the 6DOFfoot-motion data into 6-DOF motion data with the first IMU sensor as anorigin.

FIG. 5 is a schematic structural diagram of a sport training systemaccording to an embodiment of the present application. As shown in FIG.5 , the sport training system 500 comprises: a server 510, and aplurality of above-mentioned head-mounted VR devices 310 connected tothe server 510 via a network.

Each of the head-mounted VR devices 310 is configured for sending themotion data of different parts of the human body of its own userobtained by calculation to the server 510.

The server 510 is configured for receiving the motion data of thedifferent parts of the human bodies of a plurality of uses sent by theplurality of head-mounted VR devices 310; in a virtual reality scene,drawing user avatars, driving the virtual user avatars in real time byusing a graphics processing unit (GPU) on the received user data of eachof the head-mounted VR devices, and rendering and updating to obtain amulti-person interactive motion scene; and sending the multi-personinteractive motion scene to each of the head-mounted VR devices 310 inreal time.

Each of the head-mounted VR devices 310 is further configured fordisplaying the multi-person interactive motion scene on a displayscreen, so as to realize remote interaction with another sport traininguser in the virtual reality scene.

The sport training system according to the present embodiment realizesthe remote interaction among sport training users in the virtual realityscene via the server 510 and the plurality of head-mounted VR devices310 connected to the server, so that the sport training users caninteractively chat, and share fitness methods, diet precautions andother topics in the fitness process, thereby improving the interactivityand interest of fitness exercises, improving the user experience, andthus enhancing the fitness efficiency and effect.

It should be noted that the particular embodiments of the above deviceand system embodiments can be carried out with reference to theparticular embodiments of the above corresponding process embodiments,and will not be repeated here.

In sum, the technical solutions of the present application can determinewhether the user's actions are standard and whether the hold time issufficient by using merely the hand-motion tracker, the foot-motiontracker and the head-mounted VR device, and send the feedback to thehead-mounted VR device in time, and the user is not required to wear alarge number of sensors, thereby improving the user experience. Thecooperation of the head-mounted tracking camera, the electromagneticsensor/ultrasonic sensor and the IMU sensor increases the trackingprecision and frequency. The position offset error can reach the orderof millimeters, the rotation angle offset error can be less than 1.5°,the outputted frequency is greater than 200 Hz, and high-precisionhigh-frequency motion data can be obtained. Further, by comparing theobtained motion data with the demonstrative actions and marking theprompting message on the human-body part whose action is not standard,the non-demonstrative actions can be corrected in time, and variousexercise indicators such as the quantity of wrong actions, theoccurrence time of wrong actions, the average success rate of correctactions and the quantity of completed actions can be counted and fedback to the user in time, which can enable the user to understand thecompletion of the actions and improve the fitness efficiency and effect.Moreover, the remote interaction among sport training users can berealized by using the server, which further improves the userexperience.

It should be noted that:

The algorithms and displays provided herein are not inherently relevantto any specific computer, virtual device or other devices. Variousgeneral purpose devices can also be used together with the teachingherein. According to the above description, the structures that arerequired to construct this type of devices are apparent. Furthermore,the present application is not limited to any specific programminglanguage. It should be understood that the contents of the presentapplication described herein can be implemented by using variousprogramming languages, and the description above for a specific languageis intended to disclose the most preferable embodiments of the presentapplication.

The description provided herein describes many concrete details.However, it can be understood that the embodiments of the presentapplication may be implemented without those concrete details. In someof the embodiments, well-known processes, structures and techniques arenot described in detail, so as not to affect the understanding of thedescription.

Similarly, it should be understood that, in order to simplify thepresent application and help understanding one or more of the aspects ofthe present application, in the above description of the exemplaryembodiments of the present application, the features of the presentapplication are sometimes grouped into individual embodiments, figuresor the descriptions thereto. However, the method that is disclosedshould not be interpreted as reflecting the intention that the presentapplication, which is claimed, requires more feature than the featuresthat are explicitly set forth in each of the claims. More exactly, asreflected by the following claims, the aspects of the presentapplication are less than all of the features of the individualembodiments that are disclosed above. Therefore, the claims which followa particular embodiment are thereby explicitly incorporated into theparticular embodiment, and each of the claims itself serves as anseparate embodiment of the present application.

A person skilled in the art can understand that the modules in thedevice of an embodiment may be self-adaptively modified and be providedin one or more devices that are different from the embodiment. Themodules or units or components in the embodiments may be combined intoone module or unit or component, and may also be divided into multiplesubmodules or subunits or subcomponents. Unless at least some of suchfeatures and/or processes or units are mutually rejected, all of thefeatures that are disclosed by the description (including theaccompanying claims, the abstract and the drawings) and all of theprocesses or units of any method or device that is disclosed herein maybe combined in any combination. Unless explicitly stated otherwise, eachof the features that are disclosed by the description (including theaccompanying claims, the abstract and the drawings) may be replaced byalternative features that provide the same, equivalent or similarobjects.

Furthermore, a person skilled in the art can understand that, althoughsome embodiments described herein comprise a certain features that areincluded in other embodiments instead of other features, the combinationof the features of different embodiments means maintaining within thescope of the present disclosure and forming different embodiments. Forexample, in the following claims, any one of the embodiments that thepresent disclosure seeks to protect can be used in any combination.

Each component embodiment of the present application may be implementedby hardware, or by software modules that are operated on one or moreprocessors, or by a combination thereof. A person skilled in the artshould understand that some or all of the functions of some or all ofthe components of the sport training device according to the embodimentsof the present application may be implemented by using a microprocessoror a digital signal processor (DSP) in practice. The present applicationmay also be implemented as apparatus or device programs (for example,computer programs and computer program products) for implementing partof or the whole of the method described herein. Such programs forimplementing the present application may be stored in acomputer-readable medium, or may be in the form of one or more signals.Such signals may be downloaded from an Internet website, or provided ona carrier signal, or provided in any other forms.

It should be noted that the above embodiments are for describing thepresent disclosure rather than limiting the present disclosure, and aperson skilled in the art may design alternative embodiments withoutdeparting from the scope of the appended claims. In the claims, anyreference signs between parentheses should not be construed as limitingthe claims. The word “comprise” does not exclude elements or steps thatare not listed in the claims. The word “a” or “an” preceding an elementdoes not exclude the existing of a plurality of such elements. Thepresent application may be implemented by means of hardware comprisingseveral different elements and by means of a properly programmedcomputer. In unit claims that list several devices, some of thosedevices may be embodied by the same item of hardware. The words first,second, third and so on do not denote any order. Those words may beinterpreted as names.

What is claimed is:
 1. A sport training method, performed by ahead-mounted VR device wirelessly connected to a hand-motion tracker fortracking motion data of a hand of a user and a foot-motion tracker fortracking motion data of a foot of the user, wherein the methodcomprises: displaying a virtual demonstrative action; acquiring 6DOFhead-motion data, 6DOF hand-motion data and 6DOF foot-motion data of theuser, wherein acquiring the 6DOF head-motion data of the user comprises:acquiring pose information of the head-mounted VR device by using atracking camera of the head-mounted VR device; acquiring IMU informationof the head-mounted VR device by using a first IMU sensor of thehead-mounted VR device; and fusing the pose information of thehead-mounted VR device with the IMU information of the head-mounted VRdevice to obtain the 6DOF head-motion data of the user; fusing the 6DOFhand-motion data and the 6DOF foot-motion data with the 6DOF head-motiondata, comprising: acquiring a rotation matrix and a translation vectorof a built-in electromagnetic signal receiving module or ultrasonicsignal receiving module of the head-mounted VR device relative to thefirst IMU sensor, and performing coordinate system conversion on the6DOF hand-motion data and the 6DOF foot-motion data by using therotation matrix and the translation vector, to convert the 6DOFhand-motion data and the 6DOF foot-motion data into 6-DOF motion datawith the first IMU sensor as an origin; calculating motion data ofdifferent parts of a human body according to the data that have beenfused; and comparing the motion data of the different parts of the humanbody obtained by the calculation with standard sport training data ofthe virtual demonstrative action, and if a degree of similarity isgreater than a threshold, determining that an action of a human-bodypart meets a training standard, and if not, displaying a correspondingprompting message in a virtual reality scene to prompt a human-body partwhose action is not standard.
 2. The method according to claim 1,wherein the hand-motion tracker and the foot-motion tracker collect poseinformation of a hand motion and pose information of a foot motion inreal time respectively by using a built-in electromagneticsensor/ultrasonic sensor, and wirelessly transmit to an electromagneticsignal receiving module/ultrasonic signal receiving module of thehead-mounted VR device by using an electromagnetic signal emittingmodule/ultrasonic signal emitting module; the hand-motion tracker andthe foot-motion tracker also collect IMU information of the hand motionand IMU information of the foot motion in real time respectively byusing a built-in second IMU sensor, and wirelessly transmit to a firstwireless-communication module of the head-mounted VR device by using asecond wireless-communication module; acquiring the 6DOF hand-motiondata and 6DOF foot-motion data of the user comprises at least one of:receiving hand pose information and hand IMU information sent by thehand-motion tracker, and fusing the hand pose information and the handIMU information to obtain the 6DOF hand-motion data; or receiving footpose information and foot IMU information sent by the foot-motiontracker, fusing the foot pose information and the foot IMU informationto obtain the 6DOF foot-motion data.
 3. The method according to claim 1,further comprising: sending the motion data of the different parts ofthe human body obtained by the calculation to a server, so that theserver, in a virtual reality scene, according to the motion data, drawsa plurality of user avatars, and according to the plurality of useravatars, renders to generate a multi-person interactive motion scene;and receiving the multi-person interactive motion scene sent by theserver and displaying the multi-person interactive motion scene on adisplay screen, so as to realize remote interaction with another sporttraining user in the virtual reality scene.
 4. A sport training system,comprising: a server, and a plurality of head-mounted VR devicesconnected to the server via a network; each of the head-mounted VRdevices is configured for: displaying a virtual demonstrative action;acquiring 6DOF head-motion data, 6DOF hand-motion data and 6DOFfoot-motion data of a user, wherein acquiring the 6DOF head-motion dataof the user comprises: acquiring pose information of the head-mounted VRdevice by using a tracking camera of the head-mounted VR device;acquiring IMU information of the head-mounted VR device by using a firstIMU sensor of the head-mounted VR device; and fusing the poseinformation of the head-mounted VR device with the IMU information ofthe head-mounted VR device to obtain the 6DOF head-motion data of theuser; fusing the 6DOF hand-motion data and the 6DOF foot-motion datawith the 6DOF head-motion data, comprising: acquiring a rotation matrixand a translation vector of a built-in electromagnetic signal receivingmodule or ultrasonic signal receiving module of the head-mounted VRdevice relative to the first IMU sensor, and performing coordinatesystem conversion on the 6DOF hand-motion data and the 6DOF foot-motiondata by using the rotation matrix and the translation vector, to convertthe 6DOF hand-motion data and the 6DOF foot-motion data into 6-DOFmotion data with the first IMU sensor as an origin; calculating motiondata of different parts of a human body according to the data that havebeen fused; and comparing the motion data of the different parts of thehuman body obtained by the calculation with standard sport training dataof the virtual demonstrative action, and if a degree of similarity isgreater than a threshold, determining that an action of a human-bodypart meets a training standard, and if not, displaying a correspondingprompting message in a virtual reality scene to prompt a human-body partwhose action is not standard; and sending motion data of different partsof a human body obtained by calculation to the server; the server isconfigured for receiving the motion data of the different parts of thehuman body sent by the plurality of head-mounted VR devices; in avirtual reality scene, according to the motion data, drawing a pluralityof user avatars, and according to the plurality of user avatars,rendering to generate a multi-person interactive motion scene; andsending the multi-person interactive motion scene to each of thehead-mounted VR devices in real time; and each of the head-mounted VRdevices is further configured for displaying the multi-personinteractive motion scene on a display screen, so as to realize remoteinteraction with another sport training user in the virtual realityscene.
 5. The sport training system according to claim 4, wherein: thesport training system further comprises a hand-motion tracker and afoot-motion tracker configured for: collecting pose information of ahand motion and pose information of a foot motion in real timerespectively by using a built-in electromagnetic sensor/ultrasonicsensor, and wirelessly transmit to an electromagnetic signal receivingmodule/ultrasonic signal receiving module of the head-mounted VR deviceby using an electromagnetic signal emitting module/ultrasonic signalemitting module; and collecting IMU information of the hand motion andIMU information of the foot motion in real time respectively by using abuilt-in second IMU sensor, and wirelessly transmit to a firstwireless-communication module of the head-mounted VR device by using asecond wireless-communication module; the head-mounted VR devices isconfigured for: receiving hand pose information and hand IMU informationsent by the hand-motion tracker, and fusing the hand pose informationand the hand IMU information to obtain the 6DOF hand-motion data; orreceiving foot pose information and foot IMU information sent by thefoot-motion tracker, fusing the foot pose information and the foot IMUinformation to obtain the 6DOF foot-motion data.
 6. The sport trainingsystem according to claim 4, wherein the head-mounted VR devices isconfigured for: sending the motion data of the different parts of thehuman body obtained by the calculation to a server, so that the server,in a virtual reality scene, according to the motion data, draws aplurality of user avatars, and according to the plurality of useravatars, renders to generate a multi-person interactive motion scene;and receiving the multi-person interactive motion scene sent by theserver and displaying the multi-person interactive motion scene on adisplay screen, so as to realize remote interaction with another sporttraining user in the virtual reality scene.